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This  report  preser.ts  a  discussion  of  an  approach  to  building  and 
udyina  multiple  microcomputer  systems.  Tve  report  initially  discusses 
e  general  pre  hie"7  area  of  multiple  computer  systems  and  indicates  why 
search  and  development  studies  need  to  he  n u r s ' j e d  in  this   area.   This 


sort  "jres°*"ts  the  current  res: 
e  aspect   of   this  la re 


s  of  a  project  which  h~s  been  studying 

Preble"1.  This  particular  oro'ect  has  limited 


Self  t ^  the  study  of  local,  special  purpose,  multiple,  microcomputer 
chitectures.  The  particular  approach  taken  has  been  to  use  a 
neraiired  architecture  taxonomy  presented  by  Anderson  and  Jensen 
q  d  e  r  7  G  to  define  the  t y o e s  of  architectures  of  interest.  There  are 
r  architectures  in  this  taxonomy.  Each  architecture  has  distinguishing 


tributes   w  h  i  c  h 


■ill 


iharacterize   the   functional   and   ooerational 


havior   of   the   system.   The   taxonomy   only   identifies   high   level 


attributes.  Problems  associated  with  real  time  Derformance,  appropriate 
methods  to  progra™  these  architectures,  reliability  issues,  operating 
systems  Questions,  etc.  still  remain  to  be  solved.  It  is  felt  that  the! 
existence  of  an  experimental  system  whereby  any  of  the  (functionally 
representative)  Anderson  end  Jensen  architecture  types  might  be 
conveniently  built  would  provide  a  powerful  tool  to  study  some  of  these 
problems.  This  report  presents  the  current  results  of  the  project  which) 
has  been  concerned  with  designing  and  building  such  an  experimental 
microcomputer  system.  The  philosophy  followed  was  to  base  all  systems  on 
general  pupose  hardware  primitives  from  which  the  various  systems  can  be 


o  v  i 


i  "i  r 


It  is  critical  to  identify  these  general   concepts   so   that   the 


results  of  the  experimental  work  can  be  extended  to  the  general  case 
where  different  hardware  and  software  implementations  occur  and  to 
permit  considering  the  newer  VLSI  devices  which  are  becoming 
commercially  available  but  which  are  not  included  in  the  present 
experimental  system. 


Present  State  cf  Knowledge 


Advances  in  VLSI  technology  have  been  responsible  for  the 
propagation  of  digital  techniaues  for  implementing  solutions  to 
engineering  problems  which  traditionally  have  been  solved  by  other 
methods.  These  advances  have  also  made  it  possible  to  consider  building 
multicomputer  systems  where  until  quite  recently  digital  systems 
utilizing  a  single  computer  (i.e.  one  processor  in  the  system)  were  the 
norm.  These  two  trends  are  converging  so  as  to  cause  a  growth  of 
activity  in  considering  the  application  of  systems  of  multiple  computers 


in  a  variety  o*  disciplines. 

Unfortunately,  although  it  now  seems  possible  to  build  a  multiple 
computer  system,  it  is  not  clean  how  to  build  the  best  one  in  terms  of 
matching  the  design  cf  the  computer  system  to  the  needs  of  the 
application  problem.  Traditionally  the  hardware  resource  was  a  fixed 
element  and  the  software  structure  was  the  ^ajor  variable  to  consider. 
Now  the  form  o  **  the  hardware  resource  is  also  a  major  variable.  In  fact 
it  is  not  even  clear  what  types  of  mult icomputers  ~zy  be  implemented  let 
alone  making  a  rational  choice  among  well  understood  alternatives.  There 
are  a  variety  of  single  comnuters  with  different  attributes.  There  are  a 
variety  of  methods  and  mechanisms  for  interconnecting  these  single 
computers  into  multiple  computer  systems.  How  ices  the  designer  choose 
the  "best"  permutation  of  these  possibilities  as  a  solution  to  a 
particular  application  d  r  o  b  1  e  t  if  the  permutations  themselves  are  not 
well  understood?  It  is  felt  that  these  permutations  ""ay  be  described  in 
a  systematic  manner  if  the  possible  architectures  are  characterized  oy 
the  attributes  of  the  individual  com outers  comprising  the  nodes   of   the 


m u  1 1 1   compute'"   system,   'he   attributes   of   the  h a r d w a r ' 


s  of t ware 


mechanisms  used  t  ^  inter°',nr>ect  t  "r  e  individual  r  c  J  e  ^  and  *hs  attributes 
of  the  total  structure  making  uu  the  global  interconnection  subsystem. 
With  this  understanding  cf  the  behavior  and  attributes  of  the  pcssitle 
multicomputer  architectures  available  as  an  information  'base,   it   would 


then   be   f  e  a  s 


to   progress   t 


*  V,  a 


next  s  t  e 


hccsing  the  best 


architecture  for  a   <?  i  v  e  r.   problem.   With   information   available  which 
defines   multicomputer  architecture   types,  methods  may  be  developed  to 


guide  the  ger.  eratior.  of  oneratior  systems  and 


i c a t i o n  oroerams  which 


iar.    then  be  matched  to  the  underlying  hardware   structure.   Any   design 


process  involves  making  choices  among  alternatives.  Information 
describing  alternative  multicomputer  structures  is  recuirei  so  that 
these  choices  can  be  made  iv,   a  systematic  fashion. 

Tre  potential  scope  of  the  multiple  computer  architecture  space  is 
large  enough  that  a  study  cf  an  important  subset  cf  the  total  space  will 
rroviie  an  important  contribution.  An  area  of  current  interest  is 
concerned  with  special  purpose  architectures  which  are  based  upon 
microprocessors  and  compatible  VLSI  devices.  ?or  the  intents  of  this 
research  project  a  "special  purpose"  multiple  computer  architecture  is 
one  whose  characteristics  ar°  fixed  at  design  time  while  a  'general 
purpose'  architecture  is  one  which  may  be  dynamically  reconfigured  at 
run  time  according  to  the  needs  of  the  task(s)  it  is  supporting.  The 
subject  of  reliability  is  of  sufficient  importance  that  it  will  be 
considered  as  a  normal,  desirable  attribute  of  soeciai  purpose  systems. 
If  a  hardware  module  failure  occurs  reconfiguration  cf  the  system  will 
be  necessary  for  the  purpose  of  maintaining  the  system  characteristics 
which  were  specified  at  design  time.  Thus  at  least  this  aspect  of 
dynamic  reconfiguration  is  of  interest  in  special  purpose  systems.  The 
special  purpose  systems  will  encompass  the  spectrum  from  tightly  coupled 
multiprocessors  to  loosely  coupled  systems. 

?rom  a  high  level  architecture  viewpoint,  a  multiple  computer  system 
may  be  described  as  a  number  of  independent  crocessirg  nodes  which  are 
connected  together  by  an  Interconnection  Subsystem  (ISS).  Microcomputer 
cased  processing  nodes  may  be  characterized  as  conventional  general 
purpose  digital  computers  which  execute  sequential  programs.  The 
attributes  of  these  nodes  may  be  described  by  existing  notations  such  as 
the   ?MS   and   IS?   descriptive   systems   [Bell71 ,3ell78]  .   What   is  not 


presently  understood  is  how  to  effectively  describe  any  arbitrary 
rrultiple  computer  architecture.  Even  given  the  case  where  the  sane 
processing  nodes  are  'ise^  ,  a  variety  of  different  multicomputer 
architectures  are  possible  whose  characteristics  are  dependent  upon  the 
structure  of  the  ISS  which  is  employed.  An  example  of  this  is  shown  in 
firure  1.  Further  permutations  are  possible  when  it  is  considered  that 
for  a  particular  functional  ISS  tyre  there  are  a  variety  of 
implementations  which  are  dependent  upon  the  particular  hardware 
mechanisms,  and  the  associated  standards  and  protocols  which  are 
employed.  - n  example  c*  this  is  shewn  in  figure  2.  One  must  know  the 
implementation  details  for  the  ISS  before  the  operational  behavior  of 
the  multi-computer  system  can  he  definitively  described. 

The  philosophy  of  the  ?^3  arproach  to  describing  computer  systems 
is  based  upon  the  idea  that  a  fundamental  set  of  functional  building 
blocks  car  he  identified  from  which  all  digital  systems  are  comprised. 
Evidence  indicates  that  this  is  valid  for  the  uni-computer  case  and 
studies  such  as  that  which  developed  the  C-'*  architecture  [Swan'7'?] 
indicates  *hat  the  approach  is  als?  valid  for  the  multi-computer  case. 
Each  element  in  the  ?VS  set  is  a  functional  primitive.  Corresponding 
hardware  primitives  may  be  generated  reflecting  implementation 
considerations.  This  philosophy  can  apply  to  multiple  computer 
architectures.  .:  set  of  hardware  interconnection  primitives  [HI?),  along 
with  "*ec'>4?T''zei  ^tard',T'ds  and  o^ot^co"1"*  can  o*3  i  d  e  n  t  i  f  i  e  d  •  These  ^  I  ?  s 
will  form  a  primitve  set  from  which  any  ISS.  within  the  context  of 
implementations  hased  upon  VLSI,  are  realized. 

With  a  complete  set  of  hardware  building  blocks  known  for  the 
muiti -computer   case,   a   reed   exists   for  an  underlying  structure  upon 


which  to  found  the  study  of  multiple  computer  architectures.  Thus  a 
rationale  exists  for  deriving  a  taxonomy  for  multiple  computer  systems. 
£t  present  there  are  a  small  number  of  existing  taxonomies  [Ander76] 
[Free?9l  [Xro72]  t?ly?2~*  [Cren72]  .  Of  these  taxonomies  the  one  presented 
by  Anierson  and  Jensen  apoears  to  be  a  conceptually  sound  starting  point 
for  identifying  HI?s  and  ISSs.  The  nodes  in  the  tree  identify  high  level 
architecture  types  and  provide  an  initial  classification  of  ISSs.  From 
this  information  a  functionally  complete  set  of  ISSs  and  associated  HIPs 
may  te  defined . 

It  appears  that  a  variety  of  multiple  computer  configurations  are 
both  technically  a~d  economically  feasible.  However,  detailed 
information  needs  to  he  developed  which  defines  the  total  number  of 
lis tinguishable  architectures,  and  their  operational  characteristics 
before  design  decisions  can  he  nad^  in  a  systematic  manner. 


Related  research 


Distributed  system  taxonomies  have  "been  devised  by  other 
researchers  such  as  Freeman  and  Thurber  fFre°?9"!  .  Their  work  categorizes 
local  computer  networks  into  four  classes:  packet  switched,  circuit 
switched,  "bus  structure,  and  I/O  channels.  Anderson  and  Jensen  [Ander?6] 
pro  vile  a  Tore  detailed  taxonomy  with  ten  architecture  classifications. 
The  architecture  taxomony  is  based  upon  functional  attributes  of  the 
processing  elements  'the  nodes),  oaths  or  links,  ^essa^es,  ani  switches. 
Specifically  not  considered  are  communication  strategies,  message 
addressing,  cr  deadlock  situations.  The  primary  shortcoming  of  this 
taxonomy  is  that  so^e  actual  implementations  of  lis  tribute!  systems  are 
hybrids  of  the  basic  taxonomy  classes.  In  addition  the  communications 
protocol  repairs  as  an  important  unconsidered  attribute.  Thus  one 
attribute  of  the  ISS  which  can  sreatly  affect  system  performance  is  not 
included  in  the  tsaxo^o^y.  It  ~  a  y  he  ar^'ied  that  this  is  an 
implementation  consideration  rather  than  a  g e n e r a  1  system  attribute.  A 
graphical  summary  of  the  Anderson  and  Jensen  taxonomy  is  presented  in 
f  i  s  u  r  "  3 

Several  research  projects  are  underway  en  local  computer  networks. 
Of  particular  interest  are  the  definitions  of  various  hardware 
interconnection  primitives  (EI?s)  an!  interconnection  subsystems  15 3s), 
although  not  referred  to  in  those  terms.  Rennels  [Renn73]  describes  a 
fault -tolerant  distributed  system  intended  for  spacecraft.  A  number  of 
modular  building  blocks  are  defined  specifically  for  the  interconnection 
of  off-the— sel*  computer  components.  These  ire  a  memory  interface 
>I-33),  a  bus  interface  (31-33),  one  for  input-output   (IC-33),   ani   a 


so-called  "core  module"  (C0RE-BB).  Collectively  these  building  blocks 
constitute  a  class  of  ISSs,  at  least  for  the  particular  architecture 
investigated  by  Fennels.  His  redundant  global  bus  approach  may  be  more 
fault-tolerant  than  is  reouired  by  most  earth  bound  networks  tut 
self-checking  components  are  s.n    important  area  cf  interest. 

Powell  et.  ai  .  [Pow78]  iescribe  a  hybrid  architecture  with 
interesting  ISSs.  Their  network  consists  of  two  levels?  the  higher  level 
is  an  irregular  network  whose  nodes  are  interfaced  by  an  ISS  composed  of 
one  or  more  four  port  HI?s  that  collectively  make  available  an  arbitrary 
number  of  ports  to  the  network.  Only  one  of  the  ports  may  be  active  at 
ary  moment  in  time,  with  contention  control  determining  the  particular 
pert.  The  lower  level  consists  of  the  node  itself  which  implements  a 
unique  form  cf  a  bus:  a  loosely  coupled  pulse  transformer.  The  advantage 
claimed  for  such  a  HI?  is  its  fault-tolerant  properties. 

?.n  example  cf  a  Tins  topology  is  tbe  Distributed  Computer  System 
developed  at  UC  Irvine  [locmis73:.  A  control  token  is  passed  around  the 
ring  for  control  nurposes  .  Of  special  interest  is  their  definition  of  an 


nterconnecticn   primitive   ca lie' 


ocal  network  interface   or  LNI. 


This  LNI  is  in  *  er.de  d  to  also  work  in  networks  using  contention  ring 
control  and  in  contention  bus  networks  as  well.  Their  LNI  is  compose!  of 
two  HIPs?  a  network  interface  EI?  3nd  a  host-specific  HI?.  The  first  HIP 
controls  data  transmission  and  reception,  recognizes  addresses,  and 
conditions  signals.  The  host-specific  HI?  is  designed  to  exchange  data 
between  the  network  and  the  host  computer.  A  HI?  for  IMA  transfer  to  a 
LSI -11  UNIEUS  has  been  built. 

An  interesting  variation  on  the  I D D H  architecture  of   Andersen   and 
Jensen   is   the   MICP.ONET   system   [Wit?S]  .   In  this  system  each  node  is 


connected  to  a  horizontal  and  to  a  vertical  bus  through  a  HI?  defined 
for  this  system.  lach  node  consists  of  a  local  computer  (a  C)  and  a  HI? 
which  inclement s  the  bus  connections  and  which  interacts  with  the  local 
C.  In  a  given  c  or.f  igurat  ion  the  local  C  can  be  eliminated  and  a  given 
node  may  only  consist  of  the  special  HI?  which  maintains  the  ISS  at  a 
given  bus  /unction.  5n  interesting  facet  of  MIC HON ET  is  that  it 
resettles  the  IDC?,  architecture  but  it  has  an  imoortant  difference  in 
that  any  MICP.ONET  node  can  communicate  with  any  other  r.ode  connected  to 
a  bus  while  in  the  IEDR  case  a  given  node  can  only  communicate  with  a 
Neighbor  ir  the  array.  This  difference  affects  the  switching  complexity 
of  the  system,  its  reliability,  and  other  system  attributes.  Such 
differences  must  he  resolved  in  a  generally  complete  taxonomy. 

As  mentioned  earlier  a  communications  protocol  is  an  important 
aspect  of  a  distributed  system  because  of  its  infuence  upon  the  system's 
performance.  At  the  "common  carrier  network  level  much  standardization 
is  underway  by  various  international  and  national  i  n  s  t  i  t  u t ions.  A  1 o  c  a  1 
c om uu t e r  network  may  benefit  by  using  standard  components.  Standards  are 
being  discussed  at  seven  different  levels,  "any  or  "nost  of  which  ^.6.7  te 
important  to  the  context  consider0!  here.  It  is  h o p e d  that  someday  both 
hardware  and  software  modules  "ay  he  commercially  available  at 
relatively  low  cost  (compared  to  the  cost  cf  producing  "optimal"  modules 
for  a  particular  application  problem).  A  recent  technical  journal  is 
devoted  to  network  protocols  (September  1973  Computer,  Special  Issue  on 
Network  Protocols)  . 
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Current  Results  from  Project  :  Modular  Microcomputers 

An  example  cf  a  modular  microcomputer  system  is  the  Microprocessor 
Independent  Microcomputer  (MIMIC)  rCar<=\y7?a ,  Var79"!  .  This  is  based  upon 
a  standardized  mi croproc°sso r  bus,  and  a  set  of  modular  hardware 
primitives  which  are  compatible  with  any  of  the  current  generation  of  S 
bit  microprocessors.  Th°se  systems  are  being  extended  to  the 
mul  ticom"DUter  case  by  defining  hardware  primitives  from  which  a  variety 
cf  systems  may  be  realized.  A  first  set  of  hardware  in terconnec tion 
primitives  'HIP)  have  been  designee1  and  implemented  which  include  (1)  a 
shared  memory,  21  multichannel  parallel  and  serial  links,  (3)  a 
rnultiport  bus  window,  (5)  an  intelligent  channel  processor,  and  (5) 
extensions  of  the  MIMIC  bus  standard  to  support  shared  bus  systems. 
These  EI?s  are  being  installed  ir  a  three  node  MIMIC  system  so  as  to 
perform  experiments  en  a  variety  of  interconnection  subsystems  (ISS)  . 
The  selection  of  the  particular  hardware  interconnection  primitives  was 
done  by  an  analysis  of  a  high  level  taxonomy  [Ander?6] .  The  designs  were 
a  function  of  the  characteristics  of  the  MIMIC  bus  and  currently 
available  VLSI  devices.  It  is  felt  that  the  experience  gained  with  such 
a  system  will  orovide  valuable  insights  into  understanding  more  complex 
systems  based  upon  future  VLSI  devices  and  more  complex  processing  nodes 
and  interconnections  subsystems. 

*here  the  MIMIC  multicomputer  system  is  based  upon  internal 
University  of  Connecticut  designs  and  9  bit  microprocessors,  a  second 
multiple  computer  based  upon  commercial  modules,  and  the  15  bit  LSI  11 
mic ro or oces s o r ,  has  also  been  designed  and  implemented.  The 
interconnection  subsystem  (ISS)  is  entirely   based   upon   serial   links. 


11 

Although  this  is  5  very  low  pe~fornar.ee  environment  the  particular  ISS 
is  very  flexible,  permits  a  variety  of  architectures  tc  "be  implemented 
urder  nro^ran  control,  and  is  currently  be  ins  used  to  explore  problems 
with  distributed  software  systems. 

Current  Results  :  Hardware  Interconnection  Techniaues  and  Primitives 


There  are  a  limited  number  of  ways  that  digital  hardware  modules  and 
systems  ^^y  be  interconnected.  This  limitation  is  based  upon  the  nature 
cf  digital  hardware  itself.  Additional  limitations  occur  if  only  certain 
types  of  digital  systems  are  considered  suc'°  as  those  which  are  based 
upon  microprocessors  and  associated  VLSI  devices.  In  addition  to 
hardware  limits  functional  aspects  must  ~e  considered.  In  the  ?MS  system 
[3ell7l]  there  are  two  interconnection  primitives?  the  lin.-c  (L)  and  the 
switch  ^S).  This  is  a  functional  definition  and  in  practice  other 
primitives  such  as  the  memory  (v),  the  controller   £),  etc.  may  serve  to 


^^,^^0„,crjt  j|-  in  tercor,n"=rtion  function 


say   teat   something   is   an 


interconnection   mechanism  reouires  first  s  t  a  t  i  n  =r  the  level  of  viewpoint 


w h i c h   is   being  u s < 


this    caser   at    least    thre<°   viewpoint    level 


ar 


employed.   These   levels   are   the   (1)   electronic  hardware,   2^  system 
functional,  and  (3)  software  functional. 

At  the  electronic  hardware  level  in  microcomputer  basei  systems 
only  four  primary  mechanisms  exist  for  interconnecting  modules.  These 
are  'lv  bit  serial  transmission  via  a  serial  link:  under  the  control  of  a 
processor,  '2'woro1  parallel  transmission  via  a  parallel  link  under  the 
control  of  a  processor,  .3)  word  ca rail  el  transmission  ever  a  bus  [i.e. 
the  internal  microcomputer  bus),   and   (4)   word   parallel   transmission 
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through  storage  in  a  shared  memory  (I/O  devices  may  be  viewed  as  a 
snared  memory).  Thpse  ar<=  the  basic  possibilities  with  hardware 
mechansims  of  higher  functionality  being  constructed  from  this  set  of 
techniques . 

At  the  system  functional  level  the  Pm'S  concepts  appear  to  be 
complete.  Therefore  a  given  approach  will  be  characterized  as  a  ?.MS 
primitive.  An  interconnection  mechanism  may  have  a  completely 
programmable  computer  within  it  but  if  the  function  it  is  performing  is 
that  of  a  switch  then  a  switch  it  is. 

At  the  software  functional  level  the  activities  that  the  hardware 
system  is  supporting  must  be  considered.  |VcFayden  [Mcfay?6~!  suggests 
such  a  functional  viewpoint  by  proposing  the  "functional  layering" 
concent  for  organizing  software  systems  in  multicomputer  systems.  In 
this  approach  the  functional  responsibilities  of  the  network  nodes  are 
divided  into  three  major  layers  which  are  (1)  transmission  management, 
(2)  node  functional  task  management,  an>i  (3)  application  operations. 
This  identifies  at  least  three  node  tasks  which  will  have  a  time  ordered 
relationship  with  resulting  implications  about  the  potential  parallelism 
'concurrency)  of  software  operations  associated  with  the  node.  The 
performance  of  the  node  can  be  immediately  affected  by  the  hardware 
resources  in  the  node  which  are  available  to  support  (exploit)  this 
potential  parallelism. 

A  further  classification  of  the  interconnection  mechanisms  can  be 
made  if  the  system  is  divided  into  the  two  broad  classes  of  internodal 
and  intranodal  connections  with  internodal  being  those  connections 
between  disjoint  nodes  and  intranodal  being  those  connections  within  a 
distinct  node  in  the  system.  At  first  it  might  be  tempting   to   envision 
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that  fundamentally  different  hardware  mechanisms  are  necessary  to 
realize  internodal  versus  intranodal  connections.  However  this  is  not 
true.  As  stated  th°re  are  a  severely  limited  number  of  basic  digital 
hardware  possibilities  for  implement ing  hardware  connections  and  these 
cover  both  the  inter  and  intra  nodal  cases.  The  same  hardware  modules 
may  be  employed  to  realize  both.  The  distiction   becomes   a  matter   of 


functional  definti 


is  true  with  the  ?^S  primitives 


With   the   abov^   discussion   as   ar   explanation   let   the  following 
hardware  interconnection  primitives  be  defined  for   microcomputer   based 
sys  terns  . 
General  microcomputer  hardware  interconnection  primitives: 
1 .Microcomputer  h:s  'MB)  :  primary  intranodal  correction  between 

the  microprocessor  and  all  other  functional  modules  in  the  node. 
2. Microcomputer  bus  window  (MBW)  :  electronic  mapping  between  MBs  . 
3. Shared  Memory  !SM)  :  a  memory  which  may  be  accessed  by  mere  than 
one  "ricroc"r"'  outer  or  microprocessor. 

4.  Shared  Microcomputer  Pus  (S^?1  :  an  internal  bus  rfhich  may  be 
dynamically  time  shared  vy  two  rr  ~"  r e  ^icrcprccessors. 

5.  Snared  C ommuni cat  inn  Bus  :=  (SCB)  :=  an  external  bus  which  is  shared 

by  two  cr  ~cre  microcomputers  for  internodal  communication 

6.  Serial  Link  .'Is*1  :  a  bit  serial  interface  whose  operations  are 
controlled  by  a  microprocessor,  formally  only  :sed  for  transmission 
of  information. 

7. Parallel  Link  [Lpl  :  a  word  parallel  interface  whose  operations  are 
controlled  by  a  ~,icr*onT*oc::'cso"*.  May  b  ^  used  for  c  l  m  o  1  a  ^  e  s  s  a  e?  e 
transmission  or  as  part  of  a  control  -ccharism. 

£ .Transmission  Processor  (Pt)  :  a  microprocessor  whose  purpose  is  to 
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support  the  transmission  layer  within  a  node  or  between  nodes.  A 
functional  division  into  two  separate  types  of  Pt  .-nay  be  necessary 
if  internodai  and  intranodal  transmission  operations  prove  to  he 
fundamentally  different. 
9.  DMA  controller  'Kdma)  :  imolement  i.ma  intranodal  transfers. 


There   are  other  possibilities  such  as  an  alignment  network  but  upon 
examination  these  seem  to  bp  constructed  from  the  above  primitives. 

Given  these  basic  hardware  techniques  the  next  level  is  to  identify 
hardware  int  erconnect  icr.  primitives  which  may  be  used  to  realize  any 
hardware  architecture  of  interest.  In  this  study  the  Andersen  and  Jensen 
taxonomy  was  employed  to  define  such  an  architect  ire  space.  As  mentioned 


in 


introductory  discussion  other  architectures  seem  to 


s  s  i  b  1  e 


but  this  taxonomy  seems  to  be  representa ti vd  .  Therefore  the  basic  set  c * 
hardware  techniques  were  used  to  iefine  HIPs  from  which  any  of  the 
Anderson  and  Jensen  architectures  could  be  realized.  What  is  if  interest 
is  that  only  a  small  set  of  HIPs  is  reouired.  These  are  as  follows. 
PI  MIC  hardware  interconnection  orimitives  : 


link:  (SILs;4^ 


dual  mode  to  supoort  internodai 


dual  mode  to  suooort  internodai 


1 . F o  u  r   channel   s  e  r l a 
or 

intranodal  information  transfers. 
2. Four  channel  parallel  link  'S;ip;4' 
or  intranodal  information  transfers 
3.  Four  channel  bus  window  (SIBWJ41  :  address  mapping  is 

support  a  variety  of  bus  window  based  architectures. 
4. Transmission  computer  !CtTdma)  :  a  separate  computer  with  a  Lp 
and  a  Kdma  to  the  MIMIC  bus  and  which  may  control  either  the 
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SlLsJ-i  or  the  S;Lp;4:.  This  will  support  either-  internodal  ;r 

intranodal  s:ftvar=  "transmission  layers."  In  this  HI?  the  Cttdma 

will  support  either  transmission  through  a  S;LsJ4,  a  SlLp;4,  or 

it  provides  a  ima  block  transfer  capability  within  a  node.  Cost 

modularity  considerations  indicated  the  efficiency  of  grouping  these 
functi ens . 


Each  of  these  modules  has  been  designed  and  constructs 


use   m 


t  b  0  v  I M T  C  systems.  S  i  ^  c  °  th°  ^  I v  I C  s  v  s  *  ?  p  i  c  r,e^T^es°',tati7e  of  system 
such  as  the  S-100  "based  microcomputers  [31m79]  the  results  of  thi 
implementation  are  applicable  to  5-1??    type  systems  also. 

Current  Results  :  MIMIC  Mul ticomouter  Architectures 


The  following  figures  cresent  the  oarticular  L~cle~rr.  *.  a  t  i  o  n  s  of  the 
v  I M I C  H I ?  s  defined  above,  some  of  the  interconnection  subsystems 
possible  with  these  HIPs,  and  53^°  of  **~.e  resultant  multi  microcomputer 
architectures  "ii';"a  4  oresents  a  b  1  •"'  c 1,r  i  i  a  ^  r  ~  ~  o  '  * "-  =>  b  a c  i r'  four 
channel  link  c  I P  .  This  d  i  a  ^r  a rp|  represents  votv  the  5  «  L  n  *  4  and  t  b  e 
S  ;  L  s  J  4  .  F i gu r e  4  ( b  ]  and  4(c)  show  a  ?VS  iiasram  version  of  these  HIPs. 
The  ?v3  representation  is  much  clearer  ard  compact  as  a  graphical 
notation.  An  interesting  point  is  that  although  the  functional 
rep re sen tatiens  zz  the  the  S ' L ? * 4  ard  the  3  >  L  s  » 4  modules  are  the  s  a  ~  - 
their  effects  :n  the   operational   characteristics   of   the   system   r?e 


significantly   iifferent.    -s   an   eTa-ple    physical   distance   and 

transmission  speeds  are  ouite  Iifferent  and  the  S;LpJ4  nay  also  serve  in 
a  variety  of  control  operations  which  are  impossible  to  the  S*1s»4.  This 
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is  a  vivid  illustration  cf  the  need  to  examine  the  implements ti oil 
considerations  beyond  the  functional  definitions  to  see  the  important 
effects  of  usirg  different  interconnection  mechanisms. 

Figure  5  presents  a  ?^S  diagram  of  a  typical  microcomputer  such  as 
a  1*1  r"  I C  or  S-100  based  machine.  Figure  6  presents  a  ?i^S  diagram  cf  the 
transmission  processor  HI?  with  the  various  module  features  defined. 
Note  that  this  is  auite  similar  in  nature  to  the  machine  of  Figure  5  tut 
the  buffered  bus  connections  and  the  Lp  and  the  Kdma  provide  the 
hardware  means  to  realize  the  transmission  functions. 

Figure  ?  combines  the  modules  c-  figures  4,5  eid  6  to  show  a  typical 
node  which  consists  of  a  node  microcomputer  ani  the  HI?s  necessary  to 
realize  a  variety  of  interconnection  operations.  This  is  a  general 
purpose  structure  which  will  permit  implementing  a  number  cf  the 
architectures  from  the  Anderson  qv ^    Jensen  taxonomy. 

Figure  £  presents  a  block  diagram  of  the  four  channel  bus  window  and 
a  corresponding  PMS  represnta tion .  This  EI?  actually  realizes  four 
independent  bus  window  channels  in  a  single  hardware  module.  This  was 
done  for  reasons  cf  ccst  modularity  as  it  was  reasonable  to  multiplex 
the  3W  control  and  timing  functions  among  seoarate  bus  window  channels, 


Figure  9  oreserts  two  possible  architectures 


a  s  i  o :  e  * 


th  the   3WJ4 


HI?.  In  figure  9  a)  is  shown  a  three  node  configuration  which  permits 
either  a  DTI  type  machine  or  a  true  ITS  configuration.  The  address  maps 
in  the  bus  window  are  programmable  which  permits  defining  a  -a.r:oiv^  of 
bus  window  address  spaces  which  functionally  define  different 
architectures.  As  an  example  each  node  might  rave  to  ^ns^s^e  the  ring 
messages  through  a  store  and  forward  mechanism  if  the  3W  address  Tips 
only  permitted  unidirectional  information  transfer.  In  another  case  the 
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IW  air°ss  raps  may  bs  set  to  permit  any  C  in  the  network:  tc  access  any 
other  C  indirectly  through  the  EWs.  In  these  cases  the  functional 
software  operations  an^  system  "behavior  will  differ. 

Figure  10  presents  a  four  node  ring  implementation  of  the  structure 
originally  shown  in  figure  2(h).  In  this  cas-=  the  configuration  of 
figure  7.  which  employed  the  Ct  and  SJls;4  3IPs,  is  used  to  build  a 
higher  performance  DDL  architecture.  Other  basic  structures  are  also 
possible  from  this  fundamental  node. 

Present  studies  show  that  nine  of  the  ten  possible  Anderson  and 
Jensen  architectures  are  possible  from  these  hardware  primitives.  Only 
the  ICS3  structure  is  not  possible  using  these  mechanisms. 

These  systems  reflect  the  possible  multicomputer  hardware  structures 
possible  from  a  fundamental  set  of  building  blocks.  The  r. eit  phase  of 
the   study   will   define   system   level   characteristics   which   are   of 


interest 


- - 


system  attributes  will  then  be  examined  in 


of  a 


particular  EIP/ISS  implementation  to  draw  conclusions  about  the  effects 
of  particular  hardware  techniques  upon  system  level  behavior.  From  this 
information  additional  studies  will  exclore  i-velopin=j  methods  for 
matching   the   best  hardware  archit°cture  to  the  r e e d s 

croble"  . 


f  a  given  design 


urrent  Results  :  Commercial  Mult i^icrocomputer  systems 


a  here  the  M I  M I C  hardware  primitives  reflect  specialized  design   *  h  o  s  e 

primary  prupose   is   a  consideration  of  hardware  architectures,  another 

facet  of  the  project  being  presented  here  is  concerned  with  distributed 

software  issues.   It   is  not  presently  clear  what  are  the  relationships 
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■between   the   structure   of   a   distributed   software   system   and    the; 
underlying  distributed   hardware  structure.  To  study  this  and  to  permit! 
focusing  upon  the  distributed   software   issues  an     eight   node   LSIlllj 
distributed   microcomputer   has   bee",  designed  anA.    is  being  constructed.! 
This  system  has  a  general  purpose  ISS   whose   structure  may   be   varied 
under   program   control.  The  ISS  is  constructed  from  serial  link  HIPs  tai 
provide  full  point  to  point  communication  between  any  node  in  the  system 
or   any   permutation   of   interest.   The   software   system   defines   the!] 
particular   ISS   structure  which  is  to  be  used.  This  permits  an  easy  and 
flexible  shifting  between  different   fundamental   architectures   as   the 
interests  of  the  software  research  efforts  specify  a  change. 

Some  examples  of  this  system  are  shown  in  figures  12  through  12.  The  II 
basic   system   is   shown   in  figure  10  (b)  which  consists  of  eight  LSIllJ 
nodes,  as  defined  by  figure  1?  (a),  a  full  point  to   point   serial   link 
ISS,   and   a   ?DP  11/50  for  orogram  development  support.  An  example  of  a 
full  point  to  point  135  is  shown  in  figure  11  (c).  Any  rode  can  directly! 
communicate  with  any  other  node  through  dedicated  serial  links.  This   is  J 
a  completely  general  scheme  with  some  of  the  possible  permutations  shown  I 
in   figures   11    a';,   11(b),   11(d),  ll(el,  ani    12.  The  software  system] 
defines  the  dedicated  serial  links  which  it   chooses   to   use   and   that 
defines  the  ISS.  As  can    be  seen  this  oermits  TBI,  ICDS  ,  ICTL,  TDC ,  ITER, 
and   I T D I   architectures   which   are   six   of  the  ten  basic  Anderson  and 
Jensen  structures.  This  will  provide  a   great   flexibility   to   support 
experimental  work  with  distributed  software. 

To  show  other  features  of  this  system  figure  12  presents  a  dual  TLL 
system,  basically  this  is  a  system  with  an  alternate  path  to  compensate 
for   a  failure  of  a  given  dedicated  link  or  the  node  which  supports  that 
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link.  Such  a  system  provides  d.v.  experimental  vehicle  for  considering 
reliability  and  software  recovery  issues.  In  this  system  two  DDL  loops 
are  physically  defined.  Turing  nerval  operations  only  ar.  active  DDL  path 
will  he  employed.  In  case  of  a  link  failure  an  alternative  link  will  he 
activated.   In   case   of   a   node   failure   the   node   tasks   need  to  he 


tranferred  to  an  active  rode 


a      degrade!      ~iode      of      oneration      and 


associat°d  alternative  licks  will  beccme  active. 
Additional   system   a rchi t ec+ .  ures   are   rossi 
purpose  serial  link  ISS. 


■&  r»  <■>  *T> 


general 


Conclusions 

It  seems  to  be  possible  to  identify  a  set  of  architecture  types  which 
ray  be  used  to  ^ha rac feri ze  and  study  multiple  computer  systems.  It  als; 
seers  possiblD  to  identify  the  fundamental  hardware  techniques  which  arc 
employed  in  implementing  hardware  interconnection  mechanisms.  These  twi 
results  h a v e  been  combined  in  this  project  to  identify  an  initial  set  o a 
hardware  interconnection  primitives  (HI?s)  which  may  be   used   to   build 


furcticna 


^  »   ■,    ^  r  -    .         t<  o  ~i  t»  o 


sentative  interconnection  subsystems  (ISSs)  which  result 


in  a  distinguishable  architecture  tyre.  An  initial  set  of  these  3I?a 
have  been  desired  and  constructed  to  permit  the  convenient  construction 
of  the  various  multiple  microcomputer  architectures  of  interest. 

There  are  a  number  of  '•ajor  issues  which  renai:  to  be  studied  prior 
to  the  development  of  well  ordered  design  procedures  concerning  multiple 
microcomouter   systems.   Can   the  types  of  all  possible  architectures  be 


identified  and  characterized  in   a   meaningful 


*  a.y 


"  ^  i  T> 


i  s  e  n   taxonomy   indicates   t  na  t   this   is  oossible 


i  e  r  s  o  n   and 


taxonomy  is  still  necessary  which  will  also  consider  implementation 
attributes.  Can  an  encompassing  set  of  functional,  modular  primitives  be 
identified  which  s  u  o  n  c  r  t  the  implementation  3  f  a  n  v  multiple  computer 
s v s  t e m ?  This  o  r  o  i  e  c  t  has  accomplished  this  goal  within  a  limited 
context.  This  successful  result  irdioafes  that  sue-  an  obj-ctive  may  be 
generally  oossible.  The  "  ^  3  system  provides  a  conceptual  ":  a  s  e  for 
identifying   primitives   in   unicomuuters .  It  remains  to  be  seen  whether 


thi 


"  a  »>  v 


fce  ione  for  multiple  computers.  Toes  an  a  core  ornate  descriotion 


language  exist  which  can  be  used  to  iescribe,  simulate,   and  manipulate 
the   architectures   Df   multiple   computer   systems  as  a  suooort  for  the 
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design  orocess?  In  5  unicorn  outer  the  architecture  is  often  characterized 


by  either  the  structure  o 


f  y  o  ~ \  t»  n 


cesser   (stack,   register   oriented, 


etc.)  or  by  tve  processor  instruction  set  itself  [And 64,  3rown76]  .  What 
is  the  proper  way  to  characterize  the  multiple  computer  architecture? 
Perhaps  the  test  way  is  to  characterize  such  systems  in  terns  of  their 
interconnection  subsystems  which  rust  eventually  encompass  both  the 
hardware  subsystem  and  the  software  subsystem.  What  are  the  important 
attributes  of  this  overall  interconnection  subsystem?  What  are  the 
tradeoffs  between  hardware  and  software  mechanisms?  What  is  the 
relationship  between  the  distributed  hardware  system  ani  the  distributed 
software  system?  These  and  other  such  auestions  remain  to  studied  and  to 
be  answered.  It  is  felt  that  the  identification  of  the  hardware 
primitives  and  the  existence  of  the  experimental  system  iescritei  in 
this  report  will  provide  an  effective  e  x  c  e  r  i  -,  e  n.  t  a  1  m  e  a  r.  s  to  examine  sore 
of  these  issues. 
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Figure  1  (a)   An  example  of  a  high  speed 
interconnection  subsystem  using 
bus  windows  and  a  shared  memory 

System  characteristics 
Tightly  couple 
High  performance  133 
Nodes  are  physically  close 
Expensive  133 
Higher  reliability 
Good  modularity 
Central  OS  kernel 


Figure  1(b)  An  example  cf  a  low  s 
interconnection  subsystem  using 
serial  links. 

System  characteristics 


1  _ 
2. 
3 . 
a., 

5. 

6 . 

7- 
8. 


Loosely  coupled 
Low  performance  I3S 
Nodes  may  be  physically 
Inexpensive  I3S 
Low  reliability 
Good  modularity 
Distributed  OS  kernel 
etc  .  ? 
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?igure  2    (a)  DDL  architecture (Andery^) 
in  which  node  ?  must  support  both  ring 
message  transmission  and  local  tasks 


Figure  2  (b)  DDL  architecture  (Ai 
with  switch  3  supporting  ring 
message  transmission  and  node  I 

supporting  local  tasks . 
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(EXD 


a)  DDL  :  Direct 
Dedicated  Loop 


b)  DDC  :  Direct 
Dedicated  Complete 


(    C 


y 


c)  DSM  :  Direct 
Shared  Memory 


1  C*\  C*\ 


1 BUS 


d)  DSB  :  Direct 
Shared  Bus 


e)  I CDS  :  Indirect 
Centralized  Dedicated  Sta: 


f)  I  CD!  :  Indirect 
Centralized  Dedicated  loop 


I  c  )  Co 


?US 


g)  ICSB  :  Indirect 
Centralized  Shared  Bus 
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y .  I  - 

r-f  c  J —  : 
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h)    IDDR   :    Indirect 
Decentralized  Dedicated 
Regular 


. )  IDDI  :  Indirect 
decentralized  Dedi 
Irresular 


jet  „eu 


BU£ 


BW 


BUS 
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Figure  J     graphical  summary  of   the  ten  architecture  types  of  the  Anderson  and 

Jensen  taxonomy  Ar.de  r  70  . 
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Figure  }4   (a)   Block  diagram  of  two  mode, four  channel  link,  hardware 

interconnection  primitive.  Above  diagram  describes  both  serial 
and  oarallel  link  HIPs . 
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two  mode, four  channel  link  HI? 


25 


video  terminal 


L         P  „ 

serial    zoO 


M  M     r 

rom;2K     ram;62K 


M 


dual   floppy  disc 


vdisc 


microcomputer 
bus 


Connections  to  external 
systems 


Figure  5  PMS  diagram  of  a  typical  microcomputer  system 
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Figure    ,    PMS   diagram  showing   the 
a   transmission  orocessoi 


.nterconnection    of  a  node  microcomputer 
HI?,    and   a   ]4  channel   link  HI?. 
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Figure  3a   Block  diagram  of  four  channel  Bus  Window 
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Figure  8b  PMS  diagram  of 
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Figure  9(a)  Three  ncde  system  with 
Bus  Window  HIP  implementing  133. 
Possible  architectures  are  DDL  if 
window  addresses  are  set  sc  must  do 
"store  and  forward"  to  pass  messages, 
and  IDS  if  window  addresses  are  set 
so  any  C  may  access  part  of  address 
space  of  any  other  C 
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Figure  9  '"-'   Three  node  system  with 
Bus  window  HIPs  acting  to  establish  a 
DSM  architecture.  Window  addresses  sre 
set  so  each  Z   can  access  any  part  of  the 
shared  M. 
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Figure  10  Four  ncde  DDL  architecture  with  the  two  HIPs  (1)  Transmission 

Processor,  and  (2)  Pour  channel  serial  link  being  used  to  implement 

the  133.   This  is  a  specific   implementation  ;f  the  general  structure 

shown  in  figure  1  b) .  Mote  that  each  C,  1  HI?  combination  results 

t  s 
in  two  unused  serial  links  pernode.         These  may  be  usee  tc 

support  other  system  features   :r  provide  backup  links  for  reliability 

purposes  . 
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Figure  12  Dual  DDL  architecture  with  LSI11  system.  Each  connection  in  the  ring 
has  an  alternate  oath  in  case  of  a  node  failure.  Simple  extension  of  basic 
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